4-Server Infrastructure Analysis.md Quick Comparison ┌───────┬───────┬───────────┬─────┬─────┬─────┬─────┬─────┬─────┬─────────┐ │ │ │ │ │ │ │ │ Cad │ │ │ IP: 31.97.122.87 OS: Ubuntu 24.04 Cores: 2 RAM: 7.8 GB Disk %: 24% Caddy Sites: 3 Docker: 0 Stack: Static file servers ──────────────────────────────────────── Host: contabo2 Alias: contabo2 IP: 149.102.150.185 OS: Ubuntu 22.04 Cores: 8 RAM: 23 GB Disk %: 20% Caddy Sites: 18+ Docker: 11 Stack: Multi-tenant apps + PB ──────────────────────────────────────── Host: hetzner Alias: hetzner-dokploy IP: 46.62.228.173 OS: Debian 12 Cores: 8 RAM: 15 GB Disk %: 16% Caddy Sites: 17 Docker: 3 Stack: IELTS + PocketBase ──────────────────────────────────────── Host: ai-developer Alias: ai-developer IP: 213.199.36.17 OS: Ubuntu 22.04 Cores: 6 RAM: 11 GB Disk %: 34% Caddy Sites: 5 Docker: 1 Stack: Sple + OpenClaw All 4 use Caddy as the reverse proxy / TLS terminator. Caddy handles Let's Encrypt + ZeroSSL auto-issuance. --- [hostinger] 31.97.122.87 — Static Sites Proxy: Caddy (single Caddyfile, no imports) ┌─────────────────────┬────────────┬────────────────────────────┐ │ Domain │ Type │ Backend / Root │ ├─────────────────────┼────────────┼────────────────────────────┤ │ diaa.master1.vip │ static │ /var/www/taibaat │ ├─────────────────────┼────────────┼────────────────────────────┤ │ taibaat.master1.vip │ static │ /var/www/diaa-taibaat │ ├─────────────────────┼────────────┼────────────────────────────┤ │ fast.promedic1.com │ static SPA │ /var/www/fast-diet/current │ └─────────────────────┴────────────┴────────────────────────────┘ Containers: none (Docker installed but idle) Web folders: /var/www/{diaa-taibaat, fast-diet, fast-diet-releases, taibaat, taibaat-copy2} Runtime: Node v18.19.1, no PM2 --- [contabo2] 149.102.150.185 — Multi-Tenant Hub Proxy: Caddy with import sites/*.caddy (main + 18 fragments) ┌────────────────────┬────────────┬────────────────────────┐ │ Domain │ Type │ Backend │ ├────────────────────┼────────────┼────────────────────────┤ │ auto.master1.vip │ proxy │ 127.0.0.1:18789 │ ├─────────────────────┼────────────┼─────────────────────────────┤ │ banks.master1.vip │ static │ /root/banks-master1 │ ├─────────────────────┼────────────┼─────────────────────────────┤ │ bio.master1.vip │ static SPA │ /var/www/biology-exams │ ├─────────────────────┼────────────┼─────────────────────────────┤ │ bank.master1.vip │ static SPA │ /var/www/biology-banks │ ├─────────────────────┼────────────┼─────────────────────────────┤ │ math.master1.vip │ static SPA │ /var/www/math-final │ ├─────────────────────┼────────────┼─────────────────────────────┤ │ arabic.master1.vip │ static SPA │ /var/www/Arabic-banks │ ├────────────────────────┼──────────────┼─────────────────────────────┤ │ french.master1.vip │ static SPA │ /var/www/french-exams │ ├────────────────────────┼──────────────┼─────────────────────────────┤ │ eng.master1.vip │ static SPA │ /var/www/English-banks │ ├────────────────────────┼──────────────┼─────────────────────────────┤ │ physics.master1.vip │ static SPA │ /var/www/physics-bank-final │ ├────────────────────────┼──────────────┼─────────────────────────────┤ │ master1.vip │ hybrid │ PB :8090 + Telegram :8086 │ ├────────────────────────┼──────────────┼─────────────────────────────┤ │ ui.master1.vip │ proxy │ :3100 (Next.js) │ ├────────────────────────┼──────────────┼─────────────────────────────┤ │ share.master1.vip │ hybrid │ share svc :4005 + static │ ├────────────────────────┼──────────────┼─────────────────────────────┤ │ activation.master1.vip │ static + API │ PB :8090 + :8092 │ ├────────────────────────┼──────────────┼───────────────────────────────┤ │ dentist.promedic1.com │ hybrid │ Node :8085 + PB :8092 │ ├────────────────────────┼──────────────┼───────────────────────────────┤ │ umami.promedic1.com │ proxy │ localhost:3003 │ ├────────────────────────┼──────────────┼───────────────────────────────┤ │ active.addict.mom │ proxy │ :8080 │ ├────────────────────────┼──────────────┼───────────────────────────────┤ │ agent.addict.mom │ proxy │ :3006 │ ├────────────────────────┼──────────────┼───────────────────────────────┤ │ listmonk.addict.mom │ proxy │ :9000 │ ├────────────────────────┼──────────────┼───────────────────────────────┤ │ zeroclaw.addict.best │ proxy │ :18789 │ ├────────────────────────┼──────────────┼───────────────────────────────┤ │ cardx.addict.best │ static │ /var/www/cardx.addict.best │ ├────────────────────────┼──────────────┼───────────────────────────────┤ │ index.addict.best │ static │ /var/www/index.addict.best │ ├────────────────────────┼──────────────┼───────────────────────────────┤ │ duel.addict.best │ static │ /var/www/duel.addict.best │ ├────────────────────────┼──────────────┼───────────────────────────────┤ │ edugamer.addict.best │ static │ /var/www/edugamer.addict.best │ └────────────────────────┴──────────────┴───────────────────────────────┘ Containers (11): outline, outline-db, outline-redis, umami-app, umami-db, root-activepieces-1, listmonk-app, listmonk-db, root-postgres-1, root-redis-1, agent-browserless PM2 processes: dentist-ide, master1-telegram, zeroclaw-gateway PocketBase instances: :8090 (master1), :8092 (dentist) Web roots: 25+ dirs under /var/www/ (biology/chemistry/physics/math banks, master1.vip, etc.) --- [hetzner-dokploy] 46.62.228.173 — IELTS Platform Proxy: Caddy (single file, complex routes for ielts.fast) ┌────────────────────────────────────────┬─────────┬──────────────────────┐ │ Domain │ Type │ Backend │ ├────────────────────────────────────────┼─────────┼──────────────────────┤ │ │ │ PB :8090 + Node │ │ ielts.fast │ hybrid │ :8003 + TTS :5000 + │ │ │ │ Analysis :8000 + │ │ │ │ Telegram :3005 │ ├────────────────────────────────────────┼─────────┼──────────────────────┤ │ promedic1.com │ static │ /var/www/promedic1.c │ │ │ SPA │ om │ ├────────────────────────────────────────┼─────────┼──────────────────────┤ │ meals.promedic1.com │ redirec │ → diet.promedic1.com │ │ │ t │ (301) │ ├────────────────────────────────────────┼─────────┼──────────────────────┤ │ clinical.promedic1.com │ static │ /var/www/clinical │ ├────────────────────────────────────────┼─────────┼──────────────────────┤ │ drugs.promedic1.com │ — │ (route in file) │ ├────────────────────────────────────────┼─────────┼──────────────────────┤ │ diet.promedic1.com │ — │ (route in file) │ ├────────────────────────────────────────┼─────────┼──────────────────────┤ │ coach.promedic1.com │ — │ (route in file) │ ├────────────────────────────────────────┼─────────┼──────────────────────┤ │ female.promedic1.com │ — │ (route in file) │ ├────────────────────────────────────────┼─────────┼──────────────────────┤ │ listening/exercise/reading/the.ielts.f │ — │ IELTS sub-products │ │ ast │ │ │ ├────────────────────────────────────────┼─────────┼──────────────────────┤ │ pocketcoach.promedic1.com │ PB │ :8091 │ ├────────────────────────────────────────┼─────────┼──────────────────────┤ │ pocketdiet.promedic1.com │ PB │ :8092 │ ├────────────────────────────────────────┼─────────┼──────────────────────┤ │ activation.promedic1.com │ PB │ :8093 │ ├────────────────────────────────────────┼─────────┼──────────────────────┤ │ pocketbases.promedic1.com │ PB │ :8094 │ ├────────────────────────────────────────┼─────────┼──────────────────────┤ │ writing-2.ielts.fast │ — │ (route in file) │ └────────────────────────────────────────┴─────────┴──────────────────────┘ Containers (3): webapp_ielts, the-ielts, uptime-kuma PocketBase instances: ports 8090–8094 (multi-tenant) /opt stack: chatwoot, OpenHands, openhands-venv, loki, prometheus, umami Web roots: 13 dirs under /var/www/ ielts.fast has the most sophisticated routing — distinct handle blocks per API namespace (PB admin UI path rewriting, TTS, FastAPI analysis, CSP report endpoint). --- [ai-developer] 213.199.36.17 — Sple + OpenClaw Proxy: Caddy (single file) ┌───────────────────────────┬──────────┬──────────────────────────────────┐ │ Domain │ Type │ Backend │ ├───────────────────────────┼──────────┼──────────────────────────────────┤ │ prodemy.promedic1.com │ static │ /root/sple-exam-app │ │ │ SPA │ │ ├───────────────────────────┼──────────┼──────────────────────────────────┤ │ www.prodemy.promedic1.com │ redirect │ → apex (permanent) │ ├───────────────────────────┼──────────┼──────────────────────────────────┤ │ openclaw.addict.best │ proxy WS │ :18789 (websocket upgrade │ │ │ │ headers) │ ├───────────────────────────┼──────────┼──────────────────────────────────┤ │ cultures.promedic1.com │ static │ /var/www/zag-antibiogram ← zag │ │ │ SPA │ antibiogram dashboard │ ├───────────────────────────┼──────────┼──────────────────────────────────┤ │ prompting.addict.best │ proxy │ :8790 (python3 FastAPI) │ ├───────────────────────────┼──────────┼──────────────────────────────────┤ │ prompts.addict.best │ static │ /var/www/prompts-addict │ │ │ SPA │ │ └───────────────────────────┴──────────┴──────────────────────────────────┘ Containers (1): smm_automation Services: MySQL :3306, Redis :6379, openclaw :18789, python3 :8790 Web roots: /var/www/{html, prompting-navigator, prompts-addict, zag-antibiogram} Runtime: Node v24.13.1 --- Cross-Server Patterns - Caddy everywhere — consistent reverse proxy + TLS auto-renewal - master1.vip family lives on contabo2 (10+ subdomains — educational banks platform) - promedic1.com family spread across hetzner (clinical/diet/coach) + ai-developer (prodemy/cultures) + hostinger (fast) - addict.best family mostly on ai-developer + contabo2 (cardx/duel/edugamer) - PocketBase multi-tenancy concentrated on hetzner (5 instances on ports 8090–8094) - cultures.promedic1.com (the zag antibiogram worked on earlier) is on ai-developer, also accessible via zag.promedic1.com Notes / Gaps - contabo2 had 2 disabled .bak/.disabled fragments (banks.master1.vip variants) — not counted - Could not expand every route inside hetzner Caddyfile (only first 300 lines captured); the 17 listed site blocks are confirmed but their inner backends are abbreviated above - No destructive ops performed — read-only analysis Want me to drill into any one server (full route table for hetzner, full /opt inventory, container-compose mapping, etc.)? ============================================================= ⏺ Deep 4-Server Infrastructure Vision Cross-Server Health Snapshot Host: hostinger Alias: hostinger Uptime: 52d 7h Load: 0.15 RAM used: 606 Mi / 7.8 Gi Swap used: 44 Mi / 4 Gi Disk: 24% (23/96G) Kernel: 6.8.0-111 Ubuntu ──────────────────────────────────────── Host: contabo2 Alias: contabo2 Uptime: 3d 9h Load: 0.48 RAM used: 3.1 Gi / 23 Gi Swap used: 0 B / 4 Gi Disk: 20% (76/388G) Kernel: 5.15.0-185 Ubuntu ──────────────────────────────────────── Host: hetzner Alias: hetzner-dokploy Uptime: 46d 5h Load: 0.08 RAM used: 4.7 Gi / 15 Gi Swap used: 2.1 Gi / 4 Gi Disk: 16% (46/301G) Kernel: 6.1.0-48 Debian ──────────────────────────────────────── Host: ai-developer Alias: ai-developer Uptime: 127d Load: 0.98 RAM used: 1.9 Gi / 11 Gi Swap used: 0 (no swap) Disk: 34% (33/97G) Kernel: 5.15.0-170 Ubuntu Flags: hetzner is swapping hard (2.1 Gi swap used → memory pressure). ai-developer has no swap at all and highest disk use + load. Caddy versions inconsistent (2.10.2 → 2.11.2). --- [hostinger] 31.97.122.87 — Light Static Server Role: Static SPA host + dev workbench. Idle most of the time. Domains (3) ┌─────────────────────┬────────────────────────────┬───────────────────────┐ │ Domain │ Backend │ Notes │ ├─────────────────────┼────────────────────────────┼───────────────────────┤ │ diaa.master1.vip │ /var/www/taibaat │ static, caddy-owned │ ├─────────────────────┼────────────────────────────┼───────────────────────┤ │ taibaat.master1.vip │ /var/www/diaa-taibaat │ static │ ├─────────────────────┼────────────────────────────┼───────────────────────┤ │ │ │ SPA with try_files │ │ fast.promedic1.com │ /var/www/fast-diet/current │ fallback, symlinks │ │ │ │ into │ │ │ │ fast-diet-releases/ │ └─────────────────────┴────────────────────────────┴───────────────────────┘ Listening Ports :443, :80 caddy :22 sshd :2019 caddy admin API (localhost) :53 systemd-resolved No app servers, no Docker containers running, no PM2. Docker - Installed: 29.0.2 - Containers running: 0 - Networks: n8n_default (leftover), default bridge/host - Volumes: n8n_data, traefik_data (leftover from prior stack) - Images: 0 Disk Layout ┌──────────────────────────────┬───────┬────────────────────┐ │ Folder │ Size │ Files │ ├──────────────────────────────┼───────┼────────────────────┤ │ /var/www/diaa-taibaat/ │ 12 M │ 39 │ ├──────────────────────────────┼───────┼────────────────────┤ │ /var/www/fast-diet-releases/ │ 26 M │ 152 │ ├──────────────────────────────┼───────┼────────────────────┤ │ /var/www/fast-diet/ │ 4 K │ 0 (symlink target) │ ├──────────────────────────────┼───────┼────────────────────┤ │ /var/www/taibaat-copy2/ │ 1.2 M │ 6 │ ├──────────────────────────────┼───────┼────────────────────┤ │ /var/www/taibaat/ │ 84 K │ 1 │ └──────────────────────────────┴───────┴────────────────────┘ /root Workspace (248 KB metadata + 3.6 GB content) ┌────────────────────────────┬──────┬─────────────────────────────────────┐ │ Folder │ Size │ Purpose │ ├────────────────────────────┼──────┼─────────────────────────────────────┤ │ /root/output-master1/ │ 3.6 │ TikTok/Twitter/Instagram marketing │ │ │ G │ agents output │ ├────────────────────────────┼──────┼─────────────────────────────────────┤ │ /root/master1-content/ │ 165 │ content mirror │ │ │ M │ │ ├────────────────────────────┼──────┼─────────────────────────────────────┤ │ /root/تيكتوكي-فجراوي-كباس/ │ 91 M │ Arabic TikTok content │ ├────────────────────────────┼──────┼─────────────────────────────────────┤ │ /root/go/ │ 78 M │ Go toolchain │ ├────────────────────────────┼──────┼─────────────────────────────────────┤ │ /root/picoclaw/ │ 54 M │ tooling │ ├────────────────────────────┼──────┼─────────────────────────────────────┤ │ /root/video-ai-editor/ │ 172 │ script │ │ │ K │ │ ├────────────────────────────┼──────┼─────────────────────────────────────┤ │ /root/sessions/ │ 20 M │ session data │ └────────────────────────────┴──────┴─────────────────────────────────────┘ Heavy cron activity (TZ=Africa/Cairo): 6 marketing scheduler jobs every 15–60 min (TikTok ×2, Twitter, Instagram, engage_growth, weekly reset) — all targeting /root/output-master1/master1-marketing/hybrid marketing agents/hybrid-browser/. Stack: Node v18.19.1, Python 3.12.3, no PHP, no DB. --- [contabo2] 149.102.150.185 — Multi-Tenant Hub Role: The busiest server — 24 Caddy sites, 11 Docker containers, 3 PM2 apps, 2 PocketBase instances. Hosts the entire master1.vip educational platform + addict.mom/addict.best services. Domains (24 active) ┌────────────────────────┬────────────────────────────────────┬────────────┐ │ Domain │ Backend │ Type │ ├────────────────────────┼────────────────────────────────────┼────────────┤ │ auto.master1.vip │ :18789 (openclaw) │ proxy │ ├────────────────────────┼────────────────────────────────────┼────────────┤ │ banks.master1.vip │ /root/banks-master1 │ static │ ├────────────────────────┼────────────────────────────────────┼────────────┤ │ activation.master1.vip │ /root/activation.master1 + PB │ static+API │ │ │ :8090 + :8092 │ │ ├────────────────────────┼────────────────────────────────────┼────────────┤ │ master1.vip │ PB :8090 + Telegram :8086 │ hybrid │ ├────────────────────────┼────────────────────────────────────┼────────────┤ │ ui.master1.vip │ :3100 (Next.js in Docker) │ proxy │ ├────────────────────────┼────────────────────────────────────┼────────────┤ │ share.master1.vip │ share :4005 + /var/www/master1.vip │ hybrid │ ├────────────────────────┼────────────────────────────────────┼────────────┤ │ bio.master1.vip │ /var/www/biology-exams │ SPA │ ├────────────────────────┼────────────────────────────────────┼────────────┤ │ bank.master1.vip │ /var/www/biology-banks │ SPA │ ├────────────────────────┼────────────────────────────────────┼────────────┤ │ math.master1.vip │ /var/www/math-final │ SPA │ ├────────────────────────┼────────────────────────────────────┼────────────┤ │ arabic.master1.vip │ /var/www/Arabic-banks │ SPA │ ├────────────────────────┼────────────────────────────────────┼────────────┤ │ french.master1.vip │ /var/www/french-exams │ SPA │ │ french.master1.vip │ /var/www/french-exams │ SPA │ ├─────────────────────┼───────────────────────────┼──────────────────────┤ │ eng.master1.vip │ /var/www/English-banks │ SPA │ ├─────────────────────┼───────────────────────────┼──────────────────────┤ │ physics.master1.vip │ /var/www/physics-bank-fin │ SPA │ │ │ al │ │ ├─────────────────────┼───────────────────────────┼──────────────────────┤ │ dentist.promedic1.c │ Node :8085 + PB :8092 │ hybrid │ │ om │ │ │ ├─────────────────────┼───────────────────────────┼──────────────────────┤ │ umami.promedic1.com │ :3003 (Docker) │ proxy │ ├─────────────────────┼───────────────────────────┼──────────────────────┤ │ active.addict.mom │ :8080 (activepieces) │ proxy │ ├─────────────────────┼───────────────────────────┼──────────────────────┤ │ agent.addict.mom │ :3006 (browserless) │ proxy │ ├─────────────────────┼───────────────────────────┼──────────────────────┤ │ listmonk.addict.mom │ :9000 (Docker) │ proxy │ ├─────────────────────┼───────────────────────────┼──────────────────────┤ │ zeroclaw.addict.bes │ :18789 (openclaw) │ proxy │ │ t │ │ │ ├─────────────────────┼───────────────────────────┼──────────────────────┤ │ cardx.addict.best │ /var/www/cardx.addict.bes │ static │ │ │ t │ │ ├─────────────────────┼───────────────────────────┼──────────────────────┤ │ index.addict.best │ /var/www/index.addict.bes │ static │ │ │ t │ │ ├─────────────────────┼───────────────────────────┼──────────────────────┤ │ duel.addict.best │ /var/www/duel.addict.best │ static (thanwy/qodra │ │ │ │ t/tawjihy) │ ├─────────────────────┼───────────────────────────┼──────────────────────┤ │ edugamer.addict.bes │ /var/www/edugamer.addict. │ static │ │ t │ best │ │ └─────────────────────┴───────────────────────────┴──────────────────────┘ Listening Ports :443, :80 caddy :22 sshd :2019 caddy admin :18789,:18791 openclaw (PM2 zeroclaw-gateway) :3003 umami-app (docker-proxy) :3006 browserless (docker-proxy) :3100 outline (docker-proxy) :4005 master1-share (node) :8000 node app :8080 activepieces (docker-proxy) :8085 dentist-ide (node + PM2) :8086 master1-telegram (PM2) :8090 pocketbase (master1) :8092 pocketbase (dentist) :9000 listmonk (docker-proxy) Docker (11 containers / 5 compose projects / 7 networks) ┌──────────────┬──────────────────────────────────────┬────────┬─────────┐ │ Container │ Image │ Compose │ Ports │ ├─────────────────┼─────────────────────────────────┼───────────┼────────┤ │ outline │ outlinewiki/outline:latest │ logseq │ :3100 │ ├─────────────────┼─────────────────────────────────┼───────────┼────────┤ │ outline-db │ postgres:14 │ logseq │ intern │ │ │ │ │ al │ ├─────────────────┼─────────────────────────────────┼───────────┼────────┤ │ outline-redis │ redis:6 │ logseq │ intern │ │ │ │ │ al │ ├─────────────────┼─────────────────────────────────┼───────────┼────────┤ │ umami-app │ umami-software/umami:postgresql │ umami │ :3003 │ │ │ -latest │ │ │ ├─────────────────┼─────────────────────────────────┼───────────┼────────┤ │ umami-db │ postgres:15-alpine │ umami │ intern │ │ │ │ │ al │ ├─────────────────┼─────────────────────────────────┼───────────┼────────┤ │ root-activepiec │ activepieces/activepieces:lates │ activepie │ :8080 │ │ es-1 │ t │ ces │ │ ├─────────────────┼─────────────────────────────────┼───────────┼────────┤ │ root-postgres-1 │ postgres:15-alpine │ root │ intern │ │ │ │ │ al │ ├─────────────────┼─────────────────────────────────┼───────────┼────────┤ │ root-redis-1 │ redis:7-alpine │ root │ intern │ │ │ │ │ al │ ├─────────────────┼─────────────────────────────────┼───────────┼────────┤ │ listmonk-app │ listmonk/listmonk:latest │ listmonk │ :9000 │ ├─────────────────┼─────────────────────────────────┼───────────┼────────┤ │ listmonk-db │ postgres:16-alpine │ listmonk │ intern │ │ │ │ │ al │ ├─────────────────┼─────────────────────────────────┼───────────┼────────┤ │ agent-browserle │ browserless/chrome:latest │ browserle │ :3006 │ │ ss │ │ ss │ │ └─────────────────┴─────────────────────────────────┴───────────┴────────┘ Networks: activepieces_net, browserless_default, listmonk_listmonk-network, logseq_default, umami_umami-network, bridge, host, none Volumes: caddy_config, caddy_data, grafana-data, logseq_outline-data, logseq_postgres-data, logseq_redis-data, umami-db, umami_umami-postgres-data + 1 anonymous PM2 Processes ┌──────────────────┬─────────┬─────────┬────────────────────────┐ │ Name │ Mode │ Mem │ Status │ ├──────────────────┼─────────┼─────────┼────────────────────────┤ │ dentist-ide │ fork │ 64.5 MB │ online 3d │ ├──────────────────┼─────────┼─────────┼────────────────────────┤ │ master1-telegram │ cluster │ 60.5 MB │ online 3d │ ├──────────────────┼─────────┼─────────┼────────────────────────┤ │ zeroclaw-gateway │ fork │ 79.5 MB │ online 3d (5 restarts) │ ├──────────────────┼─────────┼─────────┼────────────────────────┤ │ pm2-logrotate │ module │ 69.8 MB │ online │ └──────────────────┴─────────┴─────────┴────────────────────────┘ Systemd App Services biology-exams.service, master1-share.service, pocketbase.service, pocketbase-dentist.service, pm2-root.service, fail2ban Web Roots (26 dirs) ┌─────────────────────────────────┬───────┬────────┐ │ Folder │ Size │ Files │ ├─────────────────────────────────┼───────┼────────┤ │ /var/www/z-ai-apps/ │ 346 M │ 62 150 │ ├─────────────────────────────────┼───────┼────────┤ │ /var/www/master1.vip/ │ 12 M │ 240 │ │ /var/www/master1.vip/ │ 12 M │ 240 │ ├──────────────────────────────────────────────────────────┼──────┼───────┤ │ /var/www/chemistry.master1.vip/ │ 908 │ 41 │ │ │ K │ │ ├──────────────────────────────────────────────────────────┼──────┼───────┤ │ /var/www/hero-master1/ │ 3.4 │ 8 │ │ │ M │ │ ├──────────────────────────────────────────────────────────┼──────┼───────┤ │ /var/www/dentist.promedic1.com/ │ 1.6 │ 82 │ │ │ M │ │ ├──────────────────────────────────────────────────────────┼──────┼───────┤ │ /var/www/math-zoro/ │ 1.4 │ 7 │ │ │ M │ │ ├──────────────────────────────────────────────────────────┼──────┼───────┤ │ /var/www/chem-final/ │ 648 │ 31 │ │ │ K │ │ ├──────────────────────────────────────────────────────────┼──────┼───────┤ │ /var/www/math-final/ │ 676 │ 29 │ │ │ K │ │ ├──────────────────────────────────────────────────────────┼──────┼───────┤ │ /var/www/biology-exams/ │ 360 │ 13 │ │ │ K │ │ ├──────────────────────────────────────────────────────────┼──────┼───────┤ │ /var/www/index.addict.best/ │ 988 │ 3 │ │ │ K │ │ ├──────────────────────────────────────────────────────────┼──────┼───────┤ │ others (biology-banks, English-banks, Arabic-banks, │ <1 M │ │ │ french-exams, physics-bank-final, kimi-general, │ │ — │ │ biology-egypt-legacy, physics-banks-legacy, math-zoro, │ each │ │ │ chemistry, French-oppus, edugamer, cardx, duel, html) │ │ │ └──────────────────────────────────────────────────────────┴──────┴───────┘ /opt Inventory ┌───────────────────────────────────────────────────┬───────┬─────────────┐ │ Folder │ Size │ Purpose │ ├───────────────────────────────────────────────────┼───────┼─────────────┤ │ /opt/google/ │ 403 M │ Google │ │ │ │ Cloud SDK │ ├───────────────────────────────────────────────────┼───────┼─────────────┤ │ /opt/pocketbase/ │ 61 M │ master1 PB │ │ │ │ binary │ ├───────────────────────────────────────────────────┼───────┼─────────────┤ │ /opt/dentist-pocketbase/ │ 45 M │ dentist PB │ │ │ │ binary │ ├───────────────────────────────────────────────────┼───────┼─────────────┤ │ /opt/python-tools/ │ 62 M │ python │ │ │ │ tooling │ ├───────────────────────────────────────────────────┼───────┼─────────────┤ │ /opt/kuwait-master/ │ 2.1 M │ deployment │ ├───────────────────────────────────────────────────┼───────┼─────────────┤ │ /opt/grafana-influxdb/, /opt/listmonk/, │ │ │ │ /opt/activepieces/, /opt/server-monitor/, │ small │ misc │ │ /opt/steel-browser/, /opt/rovo-work/, │ │ │ │ /opt/arabic-exams/ │ │ │ └───────────────────────────────────────────────────┴───────┴─────────────┘ /root Highlights (4.4 GB used) - /root/output-master1/ 2.4 G — marketing content output - /root/zeroclaw/ 780 M — zeroclaw backend - /root/master1-content/ 165 M - /root/dentist-unified/ 92 M - /root/تيكتوكي-فجراوي-كباس/ 91 M - /root/master1.vip/ 82 M - /root/listmonk/ 48 M - /root/zeroclaw-native/ 39 M Docs galore in /root/: 6-5-server-149-backup.md, FINAL_SUMMARY.md, ACTIVATION_CODES_SUMMARY.md, DOMAIN_STATUS_REPORT.md, GAMES_FINAL_ANALYSIS.md, CHANGE_PASSWORD_GUIDE.md. SSL: 33 certs via Caddy's ACME (Let's Encrypt + ZeroSSL). Stack: Node v20.20.0, Python 3.10.12, no PHP, no MySQL (Postgres in containers only). Cron: Same 6-job marketing scheduler pattern as hostinger (TZ=Africa/Cairo). --- [hetzner-dokploy] 46.62.228.173 — IELTS + Medical Platform Role: Premium product stack — IELTS prep platform, medical apps (promedic1 family), multi-tenant PocketBase, monitoring (Loki/Prometheus), OpenHands. Under memory pressure. Domains (18) ┌────────────────────────┬───────────────────────────────────┬────────────┐ │ Domain │ Backend │ Notes │ ├────────────────────────┼───────────────────────────────────┼────────────┤ │ │ PB :8090 + Node :8003 + TTS :5000 │ flagship, │ │ ielts.fast │ + Analysis :8000 + Telegram │ complex │ │ │ :3005 │ routes │ ├────────────────────────┼───────────────────────────────────┼────────────┤ │ the.ielts.fast │ the-ielts container :8094 │ proxy │ ├────────────────────────┼───────────────────────────────────┼────────────┤ │ │ │ IELTS │ │ writing-2.ielts.fast │ — │ Writing │ │ │ │ Task 2 │ ├───────────────────────────┼────────────────────────────────┼─────────────┤ │ listening.ielts.fast │ /var/www/ielts-listening-exams │ static │ │ │ (384 M, 26 K files) │ │ ├───────────────────────────┼────────────────────────────────┼─────────────┤ │ exercise.ielts.fast │ /var/www/ielts-exercise (764 │ static │ │ │ M, 36 K files) │ │ ├───────────────────────────┼────────────────────────────────┼─────────────┤ │ reading.ielts.fast │ /var/www/ielts-reading │ static │ ├───────────────────────────┼────────────────────────────────┼─────────────┤ │ promedic1.com │ /var/www/promedic1.com │ SPA │ ├───────────────────────────┼────────────────────────────────┼─────────────┤ │ clinical.promedic1.com │ /var/www/clinical │ medical │ │ │ │ drug ref │ │ drugs.promedic1.com │ /var/www/drugs-promedic1 │ drugs │ ├───────────────────────┼──────────────────────────────┼─────────────────┤ │ diet.promedic1.com │ PB :8092 │ PB │ │ │ (diet-pocketbase.service) │ │ ├───────────────────────┼──────────────────────────────┼─────────────────┤ │ coach.promedic1.com │ PB :8091 │ PB │ │ │ (coach-pocketbase.service) │ │ ├───────────────────────┼──────────────────────────────┼─────────────────┤ │ female.promedic1.com │ /var/www/female-promedic1 │ SPA │ ├───────────────────────┼──────────────────────────────┼─────────────────┤ │ pocketcoach.promedic1 │ PB │ sub-product │ │ .com │ │ │ ├───────────────────────┼──────────────────────────────┼─────────────────┤ │ pocketdiet.promedic1. │ PB │ sub-product │ │ com │ │ │ ├───────────────────────┼──────────────────────────────┼─────────────────┤ │ pocketbases.promedic1 │ PB │ gateway │ │ .com │ │ │ ├───────────────────────┼──────────────────────────────┼─────────────────┤ │ activation.promedic1. │ /var/www/activation.promedic │ SPA │ │ com │ 1.com (17 M) │ │ ├───────────────────────┼──────────────────────────────┼─────────────────┤ │ meals.promedic1.com │ 301 redirect │ → diet.promedic │ │ │ │ 1.com │ └───────────────────────┴──────────────────────────────┴─────────────────┘ Listening Ports :443, :80 caddy :22 sshd :2019 caddy admin :5000 python3 (TTS — public) :8000 python3/uvicorn (analysis API — public) :3002 docker → uptime-kuma :8003 docker → webapp_ielts :8090,:8091,:8092 pocketbase (ielts/coach/diet) :8094 docker → the-ielts ⚠️ :5000 (TTS) and :8000 (analysis) bind to 0.0.0.0 — directly internet-reachable, not just via Caddy. Docker (3 containers) ┌──────────────┬────────────────────────────┬───────┐ │ Container │ Image │ Ports │ ├──────────────┼────────────────────────────┼───────┤ │ webapp_ielts │ 1204ff4da8d7 │ :8003 │ ├──────────────┼────────────────────────────┼───────┤ │ the-ielts │ the-ielts-container:latest │ :8094 │ ├──────────────┼────────────────────────────┼───────┤ │ uptime-kuma │ louislam/uptime-kuma:1 │ :3002 │ └──────────────┴────────────────────────────┴───────┘ No compose project labels (manual docker run). Networks: only default bridge/host/none. Volumes (9): chatwoot_storage_data, ielts-fast-pass_app_uploads_pass, ielts-fast_app_uploads, ielts-fast_caddy_config, ielts-fast_caddy_data, ielts-fast_pb_data, ielts-fast_tts_cache, root_umami-db-data, uptime-kuma. Systemd App Services (12) ielts-analysis.service, ielts-tts.service, ielts-pocketbase.service, coach-pocketbase.service, diet-pocketbase.service, autoheal.service, caddy.service, docker.service, fail2ban.service — all running. Web Roots (13 dirs) ┌────────────────────────────────────┬───────┬────────┐ │ Folder │ Size │ Files │ ├────────────────────────────────────┼───────┼────────┤ │ /var/www/ielts-exercise/ │ 764 M │ 36 371 │ ├────────────────────────────────────┼───────┼────────┤ │ /var/www/ielts-listening-exams/ │ 384 M │ 26 329 │ ├────────────────────────────────────┼───────┼────────┤ │ /var/www/diet-plans/ │ 243 M │ 13 580 │ ├────────────────────────────────────┼───────┼────────┤ │ /var/www/drugs-promedic1/ │ 31 M │ 237 │ ├────────────────────────────────────┼───────┼────────┤ │ /var/www/activation.promedic1.com/ │ 17 M │ 4 │ ├────────────────────────────────────┼───────┼────────┤ │ /var/www/clinical/ │ 4.1 M │ 138 │ ├────────────────────────────────────┼───────┼────────┤ │ /var/www/coach.promedic1.com/ │ 4.0 M │ 41 │ ├────────────────────────────────────┼───────┼────────┤ │ /var/www/female-promedic1/ │ 1.7 M │ 50 │ ├────────────────────────────────────┼───────┼────────┤ │ /var/www/writing-task2/ │ 280 K │ 2 │ ├────────────────────────────────────┼───────┼────────┤ │ /var/www/promedic1.com/ │ 60 K │ 5 │ ├────────────────────────────────────┼───────┼────────┤ │ /var/www/ielts-reading/ │ 200 K │ 1 │ ├────────────────────────────────────┼───────┼────────┤ │ /var/www/html/ │ 120 K │ 2 │ └────────────────────────────────────┴───────┴────────┘ /opt Inventory ┌──────────────────────────────────────────┬───────┬──────────────────────┐ │ Folder │ Size │ Purpose │ ├──────────────────────────────────────────┼───────┼──────────────────────┤ │ /opt/openhands-venv/ │ 821 M │ OpenHands Python │ │ │ │ venv │ ├──────────────────────────────────────────┼───────┼──────────────────────┤ │ /opt/OpenHands/ │ 293 M │ OpenHands core │ ├──────────────────────────────────────────┼───────┼──────────────────────┤ │ /opt/chatwoot/ │ 28 K │ chatwoot config │ ├──────────────────────────────────────────┼───────┼──────────────────────┤ │ /opt/loki/, /opt/prometheus/, │ small │ monitoring configs │ │ /opt/umami/ │ │ │ └──────────────────────────────────────────┴───────┴──────────────────────┘ /root Highlights (6.5 GB used) - /root/ielts-fast/ 4.7 G — main IELTS app source - /root/female-promedic1/ 813 M - /root/trae_workoutes_restored/ 151 M - /root/diet-pocketbase/ 123 M (with backup.sh) - /root/coach-pocketbase/ 108 M (with backup.sh) - /root/promedic-tests/ 96 M - /root/ielts-pocketbase/ 68 M (with backup.sh) - /root/monitoring-agent/ 66 M - /root/nltk_data/ 49 M - /root/claude-code/ 46 M, /root/claude-skills/ 42 M Docs in /root/: AGENTS.md, ALL_APPS_AND_SERVICES.md, ALL_DOMAINS_AND_PATHS.md, APP_STABILITY_AUDIT_2026-05-05.md, APPS_ARCHITECTURE.md, COMPREHENSIVE_STABILITY_REPORT_2026-05-05.md, DEPLOYMENT_DOCUMENTATION.md, apps-uses.md, clinical_drug_summaries.json (2.1 MB), comprehensive_drug_interactions_database.md. SSL: 20 certs. Stack: Node v20.19.6, Python 3.11.2, no PHP, no MySQL. Cron (heavy ops cadence): */5 * * * * /usr/local/bin/system-alerts.sh */5 * * * * /usr/local/bin/ielts-health-watchdog.sh 0 3 * * * /root/ielts-pocketbase/backup.sh 0 2 * * * /root/coach-pocketbase/backup.sh 0 4 * * * /root/diet-pocketbase/backup.sh 0 4 * * * /usr/local/bin/restic-backup.sh 0 4 * * 0 /usr/local/bin/weekly-maintenance.sh 30 2 * * * /usr/local/bin/pb-wal-checkpoint.sh 30 4 * * * /usr/local/bin/pb-integrity-check.sh 0 6 * * 1 /root/audit-headers.sh 0 9 * * * /usr/local/bin/ielts-daily-check.sh Most rigorous ops schedule of all 4 servers — backups, integrity checks, watchdogs. --- [ai-developer] 213.199.36.17 — Apps + SMM + OpenClaw Role: Mixed: Sple exam app, OpenClaw gateway, zag antibiogram (cultures), prompting workspace, prompts library, plus SMM automation. Longest uptime (127 days). No swap → potential OOM risk. Domains (10 + 5 www redirects) ┌──────────────────────────────┬───────────────────────────┬──────────────┐ │ Domain │ Backend │ Type │ ├──────────────────────────────┼───────────────────────────┼──────────────┤ │ prodemy.promedic1.com │ /root/sple-exam-app │ SPA │ ├──────────────────────────────┼───────────────────────────┼──────────────┤ │ openclaw.addict.best │ :18789 (openclaw, WS │ proxy │ │ │ upgrade) │ │ ├──────────────────────────────┼───────────────────────────┼──────────────┤ │ cultures.promedic1.com │ /var/www/zag-antibiogram │ SPA (zag │ │ │ │ dashboard) │ ├──────────────────────────────┼───────────────────────────┼──────────────┤ │ prompting.addict.best │ :8790 (python3 FastAPI) │ proxy │ ├──────────────────────────────┼───────────────────────────┼──────────────┤ │ prompts.addict.best │ /var/www/prompts-addict │ SPA │ ├──────────────────────────────┼───────────────────────────┼──────────────┤ │ www.*.promedic1.com / │ 301 redirect │ → apex │ │ www.*.addict.best (×5) │ │ │ └──────────────────────────────┴───────────────────────────┴──────────────┘ Listening Ports :443, :80 caddy :22 sshd :2019 caddy admin :8080 caddy (secondary admin?) :18789 openclaw MainThread (IPv4+IPv6) :3306 mysqld :33060 mysqld X Protocol :6379 redis-server (IPv4+IPv6) :8790 python3 (grok-overview.service) Docker (1 container) ┌────────────────┬───────────────────────────┬────────────────┬────────────┐ │ Container │ Image │ Compose │ Ports │ ├────────────────┼───────────────────────────┼────────────────┼────────────┤ │ smm_automation │ smm_automation-automation │ smm_automation │ (internal) │ └────────────────┴───────────────────────────┴────────────────┴────────────┘ Networks: smm_automation_default, bridge, host, none Volumes: smm_automation_redis_data Images: 4 Systemd App Services grok-overview.service, caddy.service, docker.service, mysql.service, redis-server.service, php8.1-fpm.service, fail2ban.service Only server running MySQL + PHP-FPM of the 4. Web Roots (4 dirs) ┌───────────────────────────────┬───────┬───────┐ │ Folder │ Size │ Files │ ├───────────────────────────────┼───────┼───────┤ │ /var/www/zag-antibiogram/ │ 16 M │ 8 │ ├───────────────────────────────┼───────┼───────┤ │ /var/www/prompts-addict/ │ 2.2 M │ 9 │ ├───────────────────────────────┼───────┼───────┤ │ /var/www/prompting-navigator/ │ 1.2 M │ 3 │ ├───────────────────────────────┼───────┼───────┤ │ /var/www/html/ │ 8 K │ 1 │ └───────────────────────────────┴───────┴───────┘ /root Highlights (6 GB used) - /root/output-master1/ 2.3 G — marketing output (same pattern as other 3) - /root/openclaw/ 2.1 G — openclaw backend - /root/video-ai-env/ 919 M — Python venv for video AI - /root/whisper-models/ 466 M — Whisper STT models - /root/master1-content/ 206 M - /root/sessions/ 25 M - /root/maxclaw/ 16 M - /root/smm_automation/ 8.2 M - /root/hybrid-kimi-campaign/ 6.3 M - /root/facebook-campaign/ 2.4 M - /root/sple-exam-app/ 1.3 M (served via Caddy root) - /root/video-ai-editor/ 1.1 M SSL: only 1 cert (newest server config-wise). Stack: Node v24.13.1, Python 3.10.12, PHP 8.1.2, MySQL, Redis, Rust toolchain (~/.cargo, ~/.rustup), Bun. Cron: Same marketing scheduler pattern (6 jobs). Plus e2scrub_all, php. --- Cross-Cutting Observations Domain Families by Host ┌─────────────┬──────────────┬───────────────┬────────┬──────────────────┐ │ Family │ hostinger │ contabo2 │ hetzne │ ai-developer │ │ │ │ │ r │ │ ├─────────────┼──────────────┼───────────────┼────────┼──────────────────┤ │ master1.vip │ 2 (diaa/taib │ │ │ │ │ (education │ aat) │ 12 │ — │ — │ │ al) │ │ │ │ │ ├─────────────┼──────────────┼───────────────┼────────┼──────────────────┤ │ promedic1.c │ │ 2 (dentist/um │ │ 2 (prodemy/cultu │ │ om │ 1 (fast) │ ami) │ 11 │ res) │ │ (medical) │ │ │ │ │ ├─────────────┼──────────────┼───────────────┼────────┼──────────────────┤ │ ielts.fast │ — │ — │ 5 │ — │ ├─────────────┼──────────────┼───────────────┼────────┼──────────────────┤ │ addict.best │ — │ 5 │ — │ 2 │ ├─────────────┼──────────────┼───────────────┼────────┼──────────────────┤ │ addict.mom │ — │ 3 │ — │ — │ └─────────────┴──────────────┴───────────────┴────────┴──────────────────┘ Stack Summary ┌────────────┬────────────┬─────────────┬───────────────┬───────────────┐ │ Component │ hostinger │ contabo2 │ hetzner │ ai-developer │ ├────────────┼────────────┼─────────────┼───────────────┼───────────────┤ │ Caddy │ 2.11.2 │ 2.11.2 │ 2.10.2 │ 2.11.1 │ ├──────────────┼────────────┼────────────┼───────────────┼───────────────┤ │ Docker │ 29.0.2 │ 29.3.1 (11 │ 29.1.3 (3 │ 29.2.1 (1 │ │ │ (idle) │ cont.) │ cont.) │ cont.) │ ├──────────────┼────────────┼────────────┼───────────────┼───────────────┤ │ PM2 │ — │ 3 procs │ — │ — │ ├──────────────┼────────────┼────────────┼───────────────┼───────────────┤ │ PocketBase │ — │ 2 (:8090 │ 3 │ — │ │ │ │ :8092) │ (:8090-8092) │ │ ├──────────────┼────────────┼────────────┼───────────────┼───────────────┤ │ MySQL │ — │ — │ — │ active :3306 │ ├──────────────┼────────────┼────────────┼───────────────┼───────────────┤ │ Redis │ — │ docker │ docker only │ native :6379 │ │ │ │ only │ │ │ ├──────────────┼────────────┼────────────┼───────────────┼───────────────┤ │ PHP │ — │ — │ — │ 8.1 FPM │ ├──────────────┼────────────┼────────────┼───────────────┼───────────────┤ │ Node │ 18.19 │ 20.20 │ 20.19 │ 24.13 │ ├──────────────┼────────────┼────────────┼───────────────┼───────────────┤ │ Python │ 3.12 │ 3.10 │ 3.11 │ 3.10 │ ├──────────────┼────────────┼────────────┼───────────────┼───────────────┤ │ Systemd app │ 1 │ 6 │ 7 │ 4 │ │ units │ │ │ │ │ ├──────────────┼────────────┼────────────┼───────────────┼───────────────┤ │ Fail2Ban │ — │ yes │ yes │ yes │ └──────────────┴────────────┴────────────┴───────────────┴───────────────┘ Shared Patterns - Same marketing cron block runs on all 4 servers (TZ=Africa/Cairo, 6 jobs against /root/output-master1/.../hybrid-browser/) — duplicated everywhere - output-master1/ is the largest folder on every server (2.3–3.6 GB) - Caddy + ACME uniform; both LE and ZeroSSL used - OpenClaw gateway on :18789 runs on both contabo2 and ai-developer - PocketBase binary lives in /opt/pocketbase/ (contabo2) or /root/*-pocketbase/ (hetzner) Issues to Watch 1. hetzner memory pressure — 2.1 Gi swap used, 4.7 Gi / 15 Gi RAM used. OpenHands venv (821 M) + multiple PB + ielts pipeline eating RAM. 2. ai-developer no swap + 34% disk + 127-day uptime → consider reboot for kernel updates and add swap. 3. hetzner :5000 and :8000 bind 0.0.0.0 (TTS + analysis APIs) — bypass Caddy's TLS/auth. Verify firewall. 4. Caddy version drift 2.10.2 vs 2.11.x — hetzner lags. Also Caddy binary location differs (/root/.local/bin/caddy on hetzner vs /usr/bin/caddy elsewhere). 5. Duplicated marketing cron — if one server is canonical, the others are redundant runners. Confirm intent. 6. hostinger has dead Docker volumes (n8n_data, traefik_data) — leftover from prior stack, safe to prune. 7. contabo2 /var/www/z-ai-apps/ has 62 150 files in 346 M — many tiny files (probably a Node project's node_modules). Worth checking if it's stale. 8. No database backups visible on contabo2/ai-developer — hetzner has the only rigorous backup cron pattern (4 PocketBase backup jobs + restic + integrity checks). contabo2 has 11 containers with Postgres/Redis but no backup cron visible in root's crontab. Want me to drill further — full Caddyfile dumps per server, container-by-container compose file extraction, or a security audit of the :5000/:8000 exposed ports? ============================================================== Deep Dive — Caddyfiles, Compose Files, Security Audit 1. Caddyfile Deep Analysis Architecture Patterns by Server Server: hostinger Pattern: Flat single-file Sites: 3 Secrets: inline email Notes: simplest, no caching tricks ──────────────────────────────────────── Server: contabo2 Pattern: import sites/*.caddy modular Sites: 24 Secrets: env vars (.env files) Notes: most maintainable ──────────────────────────────────────── Server: hetzner Pattern: Flat single-file, monolithic Sites: 17 Secrets: /etc/caddy/secrets/pb_admin_auth import Notes: most sophisticated routing ──────────────────────────────────────── Server: ai-developer Pattern: Flat single-file Sites: 10 (5 sites + 5 www redirects) Secrets: inline Notes: cleanest site blocks Per-Server Caddy Highlights [hostinger] — 45 lines, no frills. Three static sites with identical pattern (file_server + encode zstd gzip + minimal headers). fast.promedic1.com is the only one with SPA fallback (try_files {path} /index.html) and cache-busting rules. [contabo2] — 812 lines via import sites/*.caddy. Notable patterns: - all_apps.caddy — 6 educational SPA sites in one file, all identical template (bio/bank/math/arabic/french/eng.master1.vip) - master1.vip.caddy — most advanced: blocks 15+ sensitive paths (.env, .git, composer.json, wp-*, xmlrpc.php, well-known webfinger/nodeinfo), separate CSP for /card-game* (allows 'unsafe-eval' + unpkg), service worker cache-bypass rules, query redaction in logs (format filter { wrap json fields { request>uri query { replace filter "REDACTED-FOR-PRIVACY" }}}) - share.master1.vip — hybrid: upload API to :4005, static /shared/* from disk, everything else 301→master1.vip - dentist.promedic1.com — layered: /pb/* admin PB + /api/activate + /api/notify-telegram to Node :8085, fallback to Node frontend with healthcheck - All addict.mom services have explicit robots.txt handler returning Disallow [hetzner] — 1327 lines, the most engineered: - Global block has admin 127.0.0.1:2019 (locked to localhost) + timeouts block - ielts.fast — 8 distinct route handlers in priority order: PB admin UI path rewriting (uri replace /pb/_/ /), PB API with X-Forwarded-Prefix "/pb", Telegram endpoint, TTS endpoints (matcher @tts path /api/engines /api/speak ...), Analysis endpoints, CSP report endpoint (returns 204), /api/* PB fallback. Then service worker cache rules with not path /sw.js /registerSW.js /workbox-*.js, then main app fallback to :8003 with header_down -Cache-Control (strips upstream headers so Caddy's win) - diet.promedic1.com & coach.promedic1.com — basic_auth import from /etc/caddy/secrets/pb_admin_auth for /_/* admin paths, with @adminAccess remote_ip 127.0.0.1/32 ::1/128 matcher (only localhost can even reach auth). This is real defense-in-depth. - Every site has @sensitive path /.env /.env.* /.git/* /backup.sql /backup.zip /dump.sql /phpmyadmin /wp-admin → respond @sensitive 404 - Every site has CSP with report-uri /api/csp-report; and a handle /api/csp-report { respond 204 } endpoint - the.ielts.fast → reverse_proxy :8094 (containerized the-ielts) - pocketbases.promedic1.com → static file_server from /root/pocketbases-folder (admin panel hub) - http://46.62.228.173 (bare IP) → 301 redirect to promedic1.com - All logs use format json with time_format "2006-01-02T15:04:05.000Z07:00" and rolling (50MB, keep 5, 720h retention) [ai-developer] — 259 lines, cleanest. 5 site blocks (each + www + http variants): - prodemy — SPA from /root/sple-exam-app with strict CSP (no external CDNs except cdnjs) - openclaw — reverse_proxy :18789 with explicit websocket upgrade headers (header Connection *Upgrade* / header Upgrade websocket) - cultures (zag-antibiogram) — SPA fallback, 1-hour static cache - prompting — pure proxy to python3 FastAPI :8790 - prompts — SPA from /var/www/prompts-addict - Bonus: :8080 block bound to 127.0.0.1 for local dev (prodemy served locally too) --- 2. Compose Files Inventory Only 2 servers have compose-managed stacks: contabo2 (5 projects) and ai-developer (1). hetzner containers are manual docker run. [contabo2] 5 Compose Projects ┌────────────┬────────────────────────────────┬────────────┬────────────┐ │ Project │ File │ Containers │ Resource │ │ │ │ │ Limits │ ├────────────┼────────────────────────────────┼────────────┼────────────┤ │ │ │ activepiec │ 1G/1CPU + │ Project: logseq (Outline) File: /root/logseq/docker-compose.yml Containers: outline + postgres:14 + redis:6 Resource Limits: 1G + 512M + 256M ──────────────────────────────────────── Project: umami File: /root/umami/docker-compose.yml Containers: umami + postgres:15 Resource Limits: 512M/0.5CPU + 256M/0.25CPU ──────────────────────────────────────── Project: listmonk File: /root/listmonk/docker-compose.yml Containers: listmonk + postgres:16 Resource Limits: 512M/0.5CPU + 256M/0.5CPU ──────────────────────────────────────── Project: browserless File: /root/browserless/docker-compose.yml Containers: browserless/chrome Resource Limits: 1G/1CPU Patterns observed: - All bind to 127.0.0.1: (never 0.0.0.0) — Caddy upstream-only - All have restart: unless-stopped (or always for outline) - All have explicit mem_limit + cpus (good citizen) - All have healthchecks (mostly pg_isready, redis-cli ping, wget --spider) - All have json-file logging with rotation (max-size: 50m, max-file: 5, compress: true) - All Postgres images are different versions (14, 15-alpine ×2, 16-alpine) — minor drift - Secrets use ${VAR} expansion from .env files (activepieces, umami, listmonk) - Outline has hardcoded secrets in compose (SECRET_KEY, UTILS_SECRET, POSTGRES_PASSWORD: outline) — leaked in process listing and git history potential - Volumes: logseq_* are external: true (pre-created); umami/listmonk use named volumes; activepieces uses bind mounts (/data/activepieces/...) - Networks: each project has dedicated bridge (activepieces_net, logseq_default, umami_umami-network, listmonk_listmonk-network, browserless_default) - Activepieces uses AP_EXECUTION_MODE: SANDBOXED + AP_TELEMETRY_ENABLED: "false" [ai-developer] 1 Compose Project ┌────────────────┬─────────────────────────────────────────┬───────────────┐ │ Project │ File │ Container │ ├────────────────┼─────────────────────────────────────────┼───────────────┤ │ │ │ automation │ │ smm_automation │ /root/smm_automation/docker-compose.yml │ (built from │ │ │ │ local │ │ │ │ Dockerfile) │ └────────────────┴─────────────────────────────────────────┴───────────────┘ - Built from local source (build: { context: . }) - Volumes: ./data, ./logs, ./cookies, ./screenshots (bind mounts) - Uses host.docker.internal:host-gateway to reach host Redis - Command: python main.py --mode daemon --browser adspower - No mem_limit, no healthcheck (only container, low priority) [hetzner] — No Compose 3 containers (webapp_ielts, the-ielts, uptime-kuma) all have no compose labels — launched via docker run directly. Operational gap: no IaC, harder to rebuild. [hostinger] — No Compose Docker installed but unused. Leftover volumes (n8n_data, traefik_data) from a prior stack that was torn down. --- 3. Security Audit SSH Hardening Scorecard ┌──────────────────┬────────┬──────────────┬──────────────┬──────────────┐ │ Setting │ hostin │ contabo2 │ hetzner │ ai-developer │ │ │ ger │ │ │ │ ├──────────────────┼────────┼──────────────┼──────────────┼──────────────┤ │ PermitRootLogin │ yes ⚠️ │ without-pass │ without-pass │ without-pass │ │ │ │ word │ word │ word │ ├──────────────────┼────────┼──────────────┼──────────────┼──────────────┤ │ PasswordAuthenti │ yes ⚠️ │ yes ⚠️ │ no ✓ │ no ✓ │ │ cation │ │ │ │ │ ├──────────────────┼────────┼──────────────┼──────────────┼──────────────┤ │ PubkeyAuthentica │ yes │ yes │ yes │ yes │ │ tion │ │ │ │ │ ├──────────────────┼────────┼──────────────┼──────────────┼──────────────┤ │ AllowUsers │ none │ none │ root khaled │ none │ │ filter │ │ │ ✓ │ │ ├──────────────────┼────────┼──────────────┼──────────────┼──────────────┤ │ X11Forwarding │ yes │ yes │ yes │ yes │ ├──────────────────┼────────┼──────────────┼──────────────┼──────────────┤ │ PermitEmptyPassw │ no │ no │ no │ no │ │ ords │ │ │ │ │ ├──────────────────┼────────┼──────────────┼──────────────┼──────────────┤ │ fail2ban (sshd │ not ru │ │ │ yes │ │ jail) │ nning │ yes │ yes │ (recidive │ │ │ ⚠️ │ │ │ too) │ ├──────────────────┼────────┼──────────────┼──────────────┼──────────────┤ │ fail2ban (caddy │ — │ caddy-access │ caddy-4xx │ — │ │ jail) │ │ │ │ │ ├──────────────────┼────────┼──────────────┼──────────────┼──────────────┤ │ │ │ │ not │ │ │ UFW firewall │ inacti │ active ✓ │ installed │ inactive ⚠️ │ │ │ ve ⚠️ │ │ (uses │ │ │ │ │ │ iptables) │ │ └──────────────────┴────────┴──────────────┴──────────────┴──────────────┘ Critical: hostinger is under active brute-force attack RIGHT NOW — 6+ failed root login attempts in the last 20 minutes from 178.156.137.238, and PasswordAuthentication yes + PermitRootLogin yes + no fail2ban = this is the most exploitable of the 4. Firewall Posture Server: hostinger Mechanism: none Default Policy: ACCEPT all Public Ports: 22, 80, 443 Notable Rules: — ──────────────────────────────────────── Server: contabo2 Mechanism: UFW active Default Policy: DENY incoming Public Ports: 22, 80, 443 Notable Rules: explicit DENY for 2019, 8081, 8085, 8000, 4005, 3100; ALLOW from 172.16.0.0/12 (Docker) for 2019, 9093, 9428, 8429, 3100 ──────────────────────────────────────── Server: hetzner Mechanism: raw iptables + ufw chain Default Policy: DROP (978K packets caught) Public Ports: 22, 80, 443, 5000, 8000 Notable Rules: :5000 and :8000 only accept from lo + Docker bridges (172.17.0.0/16, 172.18.0.0/16); f2b-caddy-4xx jail; f2b-sshd ──────────────────────────────────────── Server: ai-developer Mechanism: raw iptables (f2b only) Default Policy: ACCEPT Public Ports: 22, 80, 443 Notable Rules: f2b-recidive (bans repeat offenders) + f2b-sshd; handled 31M packets in f2b-recidive chain — heavy attack volume Exposed Service Audit ┌──────────────┬───────────────────────┬──────────────────────────────────┐ │ Server │ Public App Ports │ Risk │ ├──────────────┼───────────────────────┼──────────────────────────────────┤ │ hostinger │ none (only Caddy + │ Low │ │ │ SSH) │ │ ├──────────────┼───────────────────────┼──────────────────────────────────┤ │ contabo2 │ none — all app ports │ Low │ │ │ on 127.0.0.1 │ │ ├──────────────┼───────────────────────┼──────────────────────────────────┤ │ │ :5000 (TTS python3) + │ Medium — but iptables restricts │ │ hetzner │ :8000 (FastAPI │ to localhost + Docker bridges. │ │ │ uvicorn) on 0.0.0.0 │ Verify Caddy still proxies │ │ │ │ correctly. │ ├──────────────┼───────────────────────┼──────────────────────────────────┤ │ ai-developer │ none — only Caddy + │ Low │ │ │ SSH public │ │ └──────────────┴───────────────────────┴──────────────────────────────────┘ Active Attacks (last hour of auth logs) Server: hostinger Attacker IPs: 178.156.137.238 (sustained, 20+ min) Target Users: root ──────────────────────────────────────── Server: contabo2 Attacker IPs: 178.20.210.208, 2.57.121.25, 185.242.3.195 Target Users: root, admin, ubuntu ──────────────────────────────────────── Server: hetzner Attacker IPs: 31.76.49.220, 172.110.219.251, 185.100.84.174, 91.208.184.122, 45.148.10.121, 81.70.202.6, 62.210.207.172, 185.100.84.174 Target Users: user, root, deployuser, ubnt, ielts, dev ──────────────────────────────────────── Server: ai-developer Attacker IPs: 2.57.121.112, 2.57.121.25, 209.97.249.98, 192.169.232.63, 193.46.255.86, 210.1.31.3, 166.62.33.9, 77.90.185.67, 202.145.0.112, 68.183.89.16 Target Users: teagan, admin, test1, workshop, ubnt, 1hero, susan, beach hetzner & ai-developer see the highest volume (recidive counters in gigabytes) but are best protected (key-only auth + fail2ban with recidive). hostinger is the live concern. User Accounts ┌──────────────┬──────────────┐ │ Server │ Shell users │ ├──────────────┼──────────────┤ │ hostinger │ root, ubuntu │ ├──────────────┼──────────────┤ │ contabo2 │ root only │ ├──────────────┼──────────────┤ │ hetzner │ root, khaled │ ├──────────────┼──────────────┤ │ ai-developer │ root only │ └──────────────┴──────────────┘ All have %admin ALL=(ALL) ALL sudoers + 90-cloud-init-users (default cloud-init). --- Top Priority Fixes (Ranked) #: 1 Priority: CRITICAL Server: hostinger Issue: PasswordAuthentication yes + PermitRootLogin yes + active brute force + no fail2ban Fix: Edit /etc/ssh/sshd_config: set PasswordAuthentication no, PermitRootLogin prohibit-password, systemctl restart sshd. Install fail2ban: apt install -y fail2ban && systemctl enable --now fail2ban. Enable UFW: ufw allow 22/tcp && ufw allow 80/tcp && ufw allow 443/tcp && ufw --force enable. ──────────────────────────────────────── #: 2 Priority: HIGH Server: contabo2 Issue: PasswordAuthentication yes Fix: Same fix — flip to no. ──────────────────────────────────────── #: 3 Priority: MEDIUM Server: contabo2 Issue: Outline docker-compose.yml has hardcoded SECRET_KEY + UTILS_SECRET + POSTGRES_PASSWORD: outline Fix: Move to .env, rotate secrets, restart stack. ──────────────────────────────────────── #: 4 Priority: MEDIUM Server: hetzner Issue: :5000 + :8000 listen on 0.0.0.0 Fix: Edit systemd units (ielts-tts.service, ielts-analysis.service) to bind 127.0.0.1 only — current iptables rules mitigate but defense in depth wins. ──────────────────────────────────────── #: 5 Priority: MEDIUM Server: ai-developer Issue: UFW inactive (only f2b chains) Fix: ufw allow 22/tcp && ufw allow 80/tcp && ufw allow 443/tcp && ufw --force enable. ──────────────────────────────────────── #: 6 Priority: LOW Server: hostinger Issue: Stale Docker volumes n8n_data, traefik_data Fix: docker volume rm n8n_data traefik_data (free disk, remove dead state). ──────────────────────────────────────── #: 7 Priority: LOW Server: hetzner Issue: 3 containers without compose Fix: Re-create via docker-compose.yml for reproducibility. ──────────────────────────────────────── #: 8 Priority: LOW Server: all 4 Issue: Duplicated marketing cron block Fix: Verify intent — should run on 1 host, not all 4. ──────────────────────────────────────── #: 9 Priority: LOW Server: all 4 Issue: X11Forwarding yes everywhere Fix: Set to no if no X11 use case (likely none on servers). ──────────────────────────────────────── #: 10 Priority: LOW Server: hetzner Issue: Caddy 2.10.2 (others on 2.11.x) Fix: apt upgrade caddy to align versions. Want me to draft the actual remediation commands per server (so you can review before running), or apply fix #1 on hostinger immediately? ✻ Worked for 7m 28s ※ recap: Audited all 4 servers (hostinger, contabo2, hetzner, ai-developer) for domains, apps, compose stacks, and security. Found hostinger exposed: password auth + root login + no fail2ban while under active brute-force. Next: fix hostinger SSH hardening and enable UFW. ❯ for issue 1 and 5 : Keep UFW active and ensure Fail2ban is set up to work alongside it. A safe, common pattern: bash ufw default deny incoming ufw default allow outgoing ufw allow ssh ufw allow http ufw allow https ufw enable Then configure Fail2ban with action = iptables-multiport (default) so it inserts its blocks before UFW’s rules. Both will co-exist happily: UFW controls which ports are open, Fail2ban dynamically blocks bad IPs on those permitted ports. and ufw allow 22/tcp && ufw allow 80/tcp && ufw allow 443/tcp && ufw --force enable. Install and start fail2ban apt update && apt install -y fail2ban systemctl enable --now fail2ban # 3. Enable UFW with default deny ufw allow 22/tcp ufw allow 80/tcp ufw allow 443/tcp ufw --force enable 2-Make password auth entry to server to be no and depend on keys to enter. Take care when you make doors and firewalls of uwf to not block my server ports apps acess or backends, so check this carefully before allowing ufw. 3- Leave issue 7 and 8 for now , (keep them) ============================================